
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 223 13-1450 
www.uspto.gov 



APPLICATION NO. j 


FILING DATE 


FIRST NAMED INVENTOR 


ATTORNEY DOCKET NO. 


CONFIRMATION NO. 


10/741,499 


12/19/2003 


Akram A. Bou-Ghannam 


BOC9-2003-0086 (457) 


6308 



40987 7590 10/05/2007 

AKERMAN SENTERFITT 
P. O. BOX 3188 

WEST PALM BEACH, FL 33402-3188 



EXAMINER 



CAO, DIEM K 



ART UNIT 



2194 



PAPER NUMBER 



MAIL DATE 



DELIVERY MODE 



10/05/2007 PAPER 

Please find below and/or attached an Office communication concerning this application or proceeding. 

The time period for reply, if any, is set in the attached communication. 



PTOL-90A (Rev. 04/07) 



Office Action Summary 


Application No. 

10/741,499 


Applicant(s) 

BOU-GHANNAM ET AL. 


Examiner 

Diem K. Cao 


Art Unit 
2194 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )E3 Responsive to communication(s) filed on 06 July 2007 . 
2a)H This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) M Claim(s) 1-10 and 12-18 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) IEI Claim(s) 1-10,12-18 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
1 !)□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12)D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(aHd) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) Q Notice of References Cited (PTO-892) 

2) O Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) □ Information Disclosure Statement(s) (PTO/SB/08) 

Paper No(s)/Mail Date . 



super" 

4) Q Interview Summary (PTO-413) 

Paper No(s)/Mail Date. . 

5) [U Notice of Informal Patent Application 

6) □ Other: . 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 08-06) 



Office Action Summary 



Part of Paper No./Mail Date 20070922 



Application/Control Number: 10/741,499 Page 2 

Art" Unit: 2194 

DETAILED ACTION 

1. Claims 1-10 and 12-18 are pending. Applicant has amended claims 1, 2, 4-7, 9, 12, 13, 
15-18 and canceled claim 11. 

Claim Rejections - 35 USC § 102 

1 . The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

2. Claims 1-4, 6-10, 12-15 and 17-18 are rejected under 35 U.S.C. 102(e) as being 
unpatentable over Cheyer et al (U.S. 7,069,560 Bl). 

As to claim 1 , Cheyer teaches a method for managing multimodal interactions 
comprising the steps of (col. 5, lines 6-14, col. 10, lines 17-25): 

- registering a plurality of modality components with a modality component server (an 
agent registers ... vocabulary; col. 7, lines 30-34 and col. 17, lines 18-44 and Facilitator 402 is a 
specialized server agent; col. 6, lines 31-33 and 45-46 and see Figs. 3-4), wherein each modality 
component handles an interface modality for an application (a collection ... current inputs; col. 7, 
lines 13-19 and services that it provides, handler, task; col. 12, lines 19-30, lines 46-49, and col. 
17, lines 18-44), 
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- initiating a multimodal application from a client device (a mobile user . . . with standard 
interface dialog mechanism; col. 7, line 65 - col. 8, line 10), the multimodal application 
submitting activation conditions for modality components it supports to a multimodal engine of 
the modality component server (Conversely, a user may express a task to be executed by using 
typed, handwritten, or spoken English sentence; col. 8, lines 10-14), 

- activating a modality component by a modality activator of modality component server 
when activation conditions for the modality component are satisfied (If the question . . . written in 
the user interface ... natural language (NL) agent; col. 8, lines 15-19; and col. 17, lines 18-40, 
the facilitator determines the required sub-goals and then selects agents suitable for performing 
the required sub-goals; col. 18, lines 56-63), 

- connecting an activated modality component to a device (select the agents; col. 1 8, lines 
62-63 and col. 17, lines 51-59); and 

- conveying a user interaction from the device to the modality component for processing 
(when a facilitator ... facilitator; col. 7, lines 36-43; the facilitator then transmits the sub-goals 
requests to the selected agents; col. 18, lines 63-64). 

As to claim 2, Cheyer teaches the method further comprising the step of placing results 
from the user interaction onto a shared memory area of the modality component server (read or 
write shared data on the facilitator; col. 7, lines 47-50). 

As to claim 3, Cheyer teaches for each modality component, establishing a list of 
activation conditions (capabilities, triggers, tasks; col. 17, lines 18-40 and conditions; col. 22, 
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lines 21-23) such that at least one operation of the modality component is fired when one of the 
activation conditions is detected (client agent performed the requested service; col. 18, lines 36- 
45 and col. 7, lines 41-43). 

As to claim 4, Cheyer teaches wherein at least one of the plurality of modality 
components is remotely located from the device (a user interface ... sending requests to the 
facilitator ... remote application; col. 8, lines 1-14). 

As to claim 6, Cheyer teaches wherein at least one of the plurality of modality 
components is disposed within the device (user interface agent runs on the user's local laptop; 
col. 8, lines 2-4). 

As to claim 7, Cheyer teaches a modality component server comprising: 
a modality activator configured to dynamically activate at least one modality component 
responsive to an occurrence of an application event (a user interface agent; col. 7, lines 13-18 
and col. 8, line 2-7) initiated by a multimodal application from a client ( and a mobile user . . . 
with standard interface dialog mechanism ... perform the task; col. 7, line 65 - col. 8, line 14) 

- a multimodal engine including an inference engine (the facilitator ... problem solving; 
col. 6, lines 45-48), a list of activation conditions (the facilitator maintains a knowledge base that 
records the capabilities of a collection of agents; col. 16, line 66 - col. 17, line 2; capabilities, 
triggers, tasks; col. 17, lines 18-40 and conditions; col. 22, lines 21-23), and a shared memory 
(inherent from read and write shared data on the facilitator or other agent that maintain shared 
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data; col. 7, lines 45-50), the multimodal engine being configured to detect an interaction and to 
responsively initiate an interaction response by running the inference engine against the list of 
activation conditions and current state of the shared memory area (col. 16, line 66 - col. 7, line 2 
and col. 18, lines 54-66), wherein the interaction and the interaction response have been 
specified by a previously registered modality component (Facilitator agent; col. 16, line 62 - col. 
17, line 44 and col. 18, lines 54-66). 

As to claim 8, Cheyer teaches wherein a plurality of modality components are 
simultaneously utilized, wherein the plurality of modality components specify a plurality of 
interactions and associated interaction responses, and wherein the multimodal engine is 
configured to detect any of the plurality of interactions and to responsively initiate an 
programmatic action (col. 18, lines 54-66 and col. 7 5 lines 41-48) 

As to claim 9, see rejection of claim 5 above. 

As to claim 10, Cheyer teaches the multimodal engine is configured to manage 
multimodal interactions involving multiple modality components (col. 7, lines 13-18 and col. 8, 
lines 7-14 and col. 10, lines 16-25). 

As to claim 12, it is the same as the method claim of claim 1 and is rejected under the 
same ground of rejection. 
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As to claims 13-15 and 17, see rejections of claims 2-4 and 6 above. 

As to claim 1 8, it is the same as the method claim of claim 1 above and is rejected under 
the same ground of rejection. 

Claim Rejections - 35 USC §103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

4, Claims 5 and 16 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Cheyer et al (U.S. 7,069,560 Bl) in view of Pasternack et al (U.S. 6,859,451 Bl). 

As to claim 5 5 Cheyer does not explicitly teach wherein the device lacks available 
resources to locally execute at least one function that is handled by the remotely located modality 
component. However, Pasternack teaches the device lacks available resources to locally execute 
at least one function that is handled by the remotely located modality component (col. 2, lines 
53-56). It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to apply the teaching of Pasternack to the system of Cheyer because Pasternack 
provides advantages of having multimodal capability in the server rather only in the user's 
terminal include it enables advanced services to be offered to "thin" clients, and it enables new 
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capabilities to be added to services without having to distribute software to user's browser (col. 2, 
lines 49-67). 

As to claim 16, see rejection of claim 5 above. 

Response to Arguments 

5. Applicant's arguments filed 7/6/2007 have been fully considered but they are not 
persuasive. 

In the remarks, Applicant argued in substance that (1) in this instant application, the 
activation conditions for modality components supported by a client device are submitted to the 
multimodal engine and can be used by the inference engine to dynamically compare with the 
current state of the shared memory area, thus automatically activates or deactivates required 
modality component based on the user interaction without specific request by the client device. 
However, in Cheyer, the client request has to be first received and then processed though 
multiple steps in order for an appropriate agent to be found (page 11, lines 6-17). 
Examiner respectfully disagrees with Applicant's arguments: 
As to the point (1), Cheyer teaches the request includes the activation conditions, the 
multimodal engine, based on the condition and the data put on the shared memory, and the 
knowledge base to select the correct agent (multimodal component) to process the client request 
that supported the client selected multimodal input, thus, Cheyer teaches the claimed limitation. 
Furthermore, the claims in this application do not claim the request cannot be submitted prior to 
submit only the activation condition, and appropriate multimodal has been activated. 
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Therefore, the arguments are not persuasive and the rejection is maintained. 

Conclusion 

6. Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR LI 36(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Diem K. Cao whose telephone number is (571) 272-3760. The 
examiner can normally be reached on Monday - Friday, 8:30AM - 4:30PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, William Thomson can be reached on (571) 272-371 8. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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